<!DOCTYPE html>
<html id="docs" lang="en" class="">
	<head>
	<meta charset="utf-8">
<title>Running in Multiple Zones - Kubernetes</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="shortcut icon" type="image/png" href="../../../images/favicon.png">
<link rel="stylesheet" type="text/css" href="../../../css/base_fonts.css">
<link rel="stylesheet" type="text/css" href="../../../css/styles.css">
<link rel="stylesheet" type="text/css" href="https://code.jquery.com/ui/1.12.1/themes/smoothness/jquery-ui.css">
<link rel="stylesheet" type="text/css" href="https://cdnjs.cloudflare.com/ajax/libs/sweetalert/1.1.3/sweetalert.min.css">
<link rel="stylesheet" type="text/css" href="../../../css/callouts.css">
<link rel="stylesheet" type="text/css" href="../../../css/custom-jekyll/tags.css">




<meta name="description" content="Running in Multiple Zones" />
<meta property="og:description" content="Running in Multiple Zones" />

<meta property="og:url" content="https://kubernetes.io/docs/setup/multiple-zones/" />
<meta property="og:title" content="Running in Multiple Zones - Kubernetes" />

<script
src="https://code.jquery.com/jquery-3.2.1.min.js"
integrity="sha256-hwg4gsxgFZhOsEEamdOYGBf13FyQuiTwlAQgxVSNgt4="
crossorigin="anonymous"></script>
<script
src="https://code.jquery.com/ui/1.12.1/jquery-ui.min.js"
integrity="sha256-VazP97ZCwtekAsvgPBSUwPFKdrwD3unUfSGVYrahUqU="
crossorigin="anonymous"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/sweetalert/1.1.3/sweetalert.min.js"></script>
<script src="../../../js/script.js"></script>
<script src="../../../js/custom-jekyll/tags.js"></script>


	</head>
	<body>
		<div id="cellophane" onclick="kub.toggleMenu()"></div>

<header>
    <a href="../../../index.html" class="logo"></a>

    <div class="nav-buttons" data-auto-burger="primary">
        <ul class="global-nav">
            
            
            <li><a href="../../home.1">Documentation</a></li>
            
            <li><a href="../../../blog/index.html">Blog</a></li>
            
            <li><a href="../../../partners/index.html">Partners</a></li>
            
            <li><a href="../../../community/index.html">Community</a></li>
            
            <li><a href="../../../case-studies/index.html">Case Studies</a></li>
            
            
             <li>
                <a href="../multiple-zones.1#">
                    English <span class="ui-icon ui-icon-carat-1-s"></span>
                </a>
                <ul>
                
                    <li><a href="../../../zh/index.html">中文 Chinese</a></li>
                
                    <li><a href="../../../ko/index.html">한국어 Korean</a></li>
                
                </ul>
            </li>
         
            <li>
                <a href="../multiple-zones.1#">
                    v1.11 <span class="ui-icon ui-icon-carat-1-s"></span>
                </a>
                <ul>
                
                    <li><a href="https://kubernetes.io">v1.12</a></li>
                
                    <li><a href="../../../index.html">v1.11</a></li>
                
                    <li><a href="https://v1-10.docs.kubernetes.io">v1.10</a></li>
                
                    <li><a href="https://v1-9.docs.kubernetes.io">v1.9</a></li>
                
                </ul>
            </li>
        </ul>
        
        <a href="../../tutorials/kubernetes-basics/index.html" class="button" id="tryKubernetes" data-auto-burger-exclude>Try Kubernetes</a>
        <button id="hamburger" onclick="kub.toggleMenu()" data-auto-burger-exclude><div></div></button>
    </div>

    <nav id="mainNav">
        <main data-auto-burger="primary">
        <div class="nav-box">
            <h3><a href="../../tutorials/stateless-application/hello-minikube/index.html">Get Started</a></h3>
            <p>Ready to get your hands dirty? Build a simple Kubernetes cluster that runs "Hello World" for Node.js.</p>
        </div>
        <div class="nav-box">
            <h3><a href="../../home.1">Documentation</a></h3>
            <p>Learn how to use Kubernetes with the use of walkthroughs, samples, and reference documentation. You can even <a href="../../../editdocs/index.html" data-auto-burger-exclude>help contribute to the docs</a>!</p>
        </div>
        <div class="nav-box">
            <h3><a href="../../../community/index.html">Community</a></h3>
            <p>If you need help, you can connect with other Kubernetes users and the Kubernetes authors, attend community events, and watch video presentations from around the web.</p>
        </div>
        <div class="nav-box">
            <h3><a href="../../../blog/index.html">Blog</a></h3>
            <p>Read the latest news for Kubernetes and the containers space in general, and get technical how-tos hot off the presses.</p>
        </div>
        </main>
        <main data-auto-burger="primary">
        <div class="left">
            <h5 class="github-invite">Interested in hacking on the core Kubernetes code base?</h5>
            <a href="https://github.com/kubernetes/kubernetes" class="button" data-auto-burger-exclude>View On Github</a>
        </div>

        <div class="right">
            <h5 class="github-invite">Explore the community</h5>
            <div class="social">
                <a href="https://twitter.com/kubernetesio" class="twitter"><span>Twitter</span></a>
                <a href="https://github.com/kubernetes/kubernetes" class="github"><span>Github</span></a>
                <a href="http://slack.k8s.io/" class="slack"><span>Slack</span></a>
                <a href="http://stackoverflow.com/questions/tagged/kubernetes" class="stack-overflow"><span>Stack Overflow</span></a>
                <a href="https://discuss.kubernetes.io" class="mailing-list"><span>Forum</span></a>
                <a href="https://calendar.google.com/calendar/embed?src=nt2tcnbtbied3l6gi2h29slvc0%40group.calendar.google.com" class="calendar"><span>Events Calendar</span></a>
            </div>
        </div>
        <div class="clear" style="clear: both"></div>
        </main>
    </nav>
</header>

		
		
		<section id="hero" class="light-text no-sub">
			







<h1>Setup</h1>
<h5></h5>










<div id="vendorStrip" class="light-text">
	<ul>
		
		
		<li><a href="../../home.1">DOCUMENTATION</a></li>
		
		
		<li><a href="../index.html" class="YAH">SETUP</a></li>
		
		
		<li><a href="../../concepts/index.html">CONCEPTS</a></li>
		
		
		<li><a href="../../tasks/index.html">TASKS</a></li>
		
		
		<li><a href="../../tutorials/index.html">TUTORIALS</a></li>
		
		
		<li><a href="../../reference.1">REFERENCE</a></li>
		
	</ul>
	<div id="searchBox">
		<input type="text" id="search" placeholder="Search" onkeydown="if (event.keyCode==13) window.location.replace('/docs/search/?q=' + this.value)" autofocus="autofocus">
	</div>
</div>

		</section>
		
		
<section id="deprecationWarning">
  <main>
    <div class="content deprecation-warning">
      <h3>
        Documentation for Kubernetes v1.11 is no longer actively maintained. The version you are currently viewing is a static snapshot.
        For up-to-date documentation, see the <a href="https://kubernetes.io/docs/home/">latest</a> version.
      </h3>
    </div>
  </main>
</section>


		<section id="encyclopedia">
			
<div id="docsToc">
     <div class="pi-accordion">
    	
        
        
        
        
        
         
             
                 
             
         
             
                 
                          
                          
                 
             
         
             
         
             
         
             
         
             
         
             
         
             
         
         
        
        <a class="item" data-title="Setup" href="../index.html"></a>

	
	
		
		
	<div class="item" data-title="Bootstrapping Clusters with kubeadm">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Customizing control plane configuration with kubeadm" href="../independent/control-plane-flags/index.html"></a>

		
	
		
		
<a class="item" data-title="Creating Highly Available Clusters with kubeadm" href="../independent/high-availability.1"></a>

		
	
		
		
<a class="item" data-title="Creating a single master cluster with kubeadm" href="../../getting-started-guides/kubeadm/index.html"></a>

		
	
		
		
<a class="item" data-title="Troubleshooting kubeadm" href="../independent/troubleshooting-kubeadm/index.html"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="Turnkey Cloud Solutions">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Running Kubernetes on AWS EC2" href="../turnkey/aws/index.html"></a>

		
	
		
		
<a class="item" data-title="Running Kubernetes on Alibaba Cloud" href="../turnkey/alibaba-cloud/index.html"></a>

		
	
		
		
<a class="item" data-title="Running Kubernetes on Azure" href="../turnkey/azure/index.html"></a>

		
	
		
		
<a class="item" data-title="Running Kubernetes on CenturyLink Cloud" href="../turnkey/clc/index.html"></a>

		
	
		
		
<a class="item" data-title="Running Kubernetes on Google Compute Engine" href="../turnkey/gce/index.html"></a>

		
	
		
		
<a class="item" data-title="Running Kubernetes on Multiple Clouds with Stackpoint.io" href="../turnkey/stackpoint/index.html"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="On-Premises VMs">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Cloudstack" href="../../getting-started-guides/cloudstack/index.html"></a>

		
	
		
		
<a class="item" data-title="Kubernetes on DCOS" href="../../getting-started-guides/dcos/index.html"></a>

		
	
		
		
<a class="item" data-title="oVirt" href="../../getting-started-guides/ovirt.1"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="Custom Cloud Solutions">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="CoreOS on AWS or GCE" href="../../getting-started-guides/coreos.1"></a>

		
	
		
		
<a class="item" data-title="Installing Kubernetes On-premises/Cloud Providers with Kubespray" href="../../getting-started-guides/kubespray/index.html"></a>

		
	
		
		
<a class="item" data-title="Installing Kubernetes on AWS with kops" href="../custom-cloud/kops.1"></a>

		
	

		</div>
	</div>

		
	
		
		
<a class="item" data-title="Building Large Clusters" href="../cluster-large/index.html"></a>

		
	
		
		
<a class="item" data-title="Building from Source" href="../building-from-source/index.html"></a>

		
	
		
		
<a class="item" data-title="Configuring Kubernetes with Salt" href="../salt/index.html"></a>

		
	
		
		
<a class="item" data-title="Creating a Custom Cluster from Scratch" href="../../getting-started-guides/scratch/index.html"></a>

		
	
		
		
<a class="item" data-title="Picking the Right Solution" href="../../getting-started-guides/index.html"></a>

		
	
		
		
<a class="item" data-title="Running Kubernetes Locally via Minikube" href="../../getting-started-guides/minikube/index.html"></a>

		
	
		
		
<a class="item" data-title="Running in Multiple Zones" href="../multiple-zones.1"></a>

		
	
		
		
<a class="item" data-title="Validate Node Setup" href="../node-conformance/index.html"></a>

		
	






     </div> 
    <button class="push-menu-close-button" onclick="kub.toggleToc()"></button>
</div> 

			<div id="docsContent">
				
<p><a href="../../editdocs#docs/setup/multiple-zones.md" id="editPageButton">Edit This Page</a></p>

<h1>Running in Multiple Zones</h1>





<h2 id="introduction">Introduction</h2>

<p>Kubernetes 1.2 adds support for running a single cluster in multiple failure zones
(GCE calls them simply &ldquo;zones&rdquo;, AWS calls them &ldquo;availability zones&rdquo;, here we&rsquo;ll refer to them as &ldquo;zones&rdquo;).
This is a lightweight version of a broader Cluster Federation feature (previously referred to by the affectionate
nickname <a href="https://github.com/kubernetes/community/blob/v1.11.3/contributors/design-proposals/multicluster/federation.md" target="_blank">&ldquo;Ubernetes&rdquo;</a>).
Full Cluster Federation allows combining separate
Kubernetes clusters running in different regions or cloud providers
(or on-premises data centers).  However, many
users simply want to run a more available Kubernetes cluster in multiple zones
of their single cloud provider, and this is what the multizone support in 1.2 allows
(this previously went by the nickname &ldquo;Ubernetes Lite&rdquo;).</p>

<p>Multizone support is deliberately limited: a single Kubernetes cluster can run
in multiple zones, but only within the same region (and cloud provider).  Only
GCE and AWS are currently supported automatically (though it is easy to
add similar support for other clouds or even bare metal, by simply arranging
for the appropriate labels to be added to nodes and volumes).</p>

<nav id="TableOfContents">
<ul>
<li>
<ul>
<li><a href="../multiple-zones.1#introduction">Introduction</a></li>
<li><a href="../multiple-zones.1#functionality">Functionality</a></li>
<li><a href="../multiple-zones.1#limitations">Limitations</a></li>
<li><a href="../multiple-zones.1#walkthrough">Walkthrough</a>
<ul>
<li><a href="../multiple-zones.1#bringing-up-your-cluster">Bringing up your cluster</a></li>
<li><a href="../multiple-zones.1#nodes-are-labeled">Nodes are labeled</a></li>
<li><a href="../multiple-zones.1#add-more-nodes-in-a-second-zone">Add more nodes in a second zone</a></li>
<li><a href="../multiple-zones.1#volume-affinity">Volume affinity</a></li>
<li><a href="../multiple-zones.1#pods-are-spread-across-zones">Pods are spread across zones</a></li>
<li><a href="../multiple-zones.1#shutting-down-the-cluster">Shutting down the cluster</a></li>
</ul></li>
</ul></li>
</ul>
</nav>

<h2 id="functionality">Functionality</h2>

<p>When nodes are started, the kubelet automatically adds labels to them with
zone information.</p>

<p>Kubernetes will automatically spread the pods in a replication controller
or service across nodes in a single-zone cluster (to reduce the impact of
failures.)  With multiple-zone clusters, this spreading behavior is
extended across zones (to reduce the impact of zone failures.)  (This is
achieved via <code>SelectorSpreadPriority</code>).  This is a best-effort
placement, and so if the zones in your cluster are heterogeneous
(e.g. different numbers of nodes, different types of nodes, or
different pod resource requirements), this might prevent perfectly
even spreading of your pods across zones. If desired, you can use
homogeneous zones (same number and types of nodes) to reduce the
probability of unequal spreading.</p>

<p>When persistent volumes are created, the <code>PersistentVolumeLabel</code>
admission controller automatically adds zone labels to them.  The scheduler (via the
<code>VolumeZonePredicate</code> predicate) will then ensure that pods that claim a
given volume are only placed into the same zone as that volume, as volumes
cannot be attached across zones.</p>

<h2 id="limitations">Limitations</h2>

<p>There are some important limitations of the multizone support:</p>

<ul>
<li><p>We assume that the different zones are located close to each other in the
network, so we don&rsquo;t perform any zone-aware routing.  In particular, traffic
that goes via services might cross zones (even if pods in some pods backing that service
exist in the same zone as the client), and this may incur additional latency and cost.</p></li>

<li><p>Volume zone-affinity will only work with a <code>PersistentVolume</code>, and will not
work if you directly specify an EBS volume in the pod spec (for example).</p></li>

<li><p>Clusters cannot span clouds or regions (this functionality will require full
federation support).</p></li>

<li><p>Although your nodes are in multiple zones, kube-up currently builds
a single master node by default.  While services are highly
available and can tolerate the loss of a zone, the control plane is
located in a single zone.  Users that want a highly available control
plane should follow the <a href="../../admin/high-availability">high availability</a> instructions.</p></li>

<li><p>StatefulSet volume zone spreading when using dynamic provisioning is currently not compatible with
pod affinity or anti-affinity policies.</p></li>

<li><p>If the name of the StatefulSet contains dashes (&ldquo;-&rdquo;), volume zone spreading
may not provide a uniform distribution of storage across zones.</p></li>

<li><p>When specifying multiple PVCs in a Deployment or Pod spec, the StorageClass
needs to be configured for a specific, single zone, or the PVs need to be
statically provisioned in a specific zone. Another workaround is to use a
StatefulSet, which will ensure that all the volumes for a replica are
provisioned in the same zone.</p></li>
</ul>

<h2 id="walkthrough">Walkthrough</h2>

<p>We&rsquo;re now going to walk through setting up and using a multi-zone
cluster on both GCE &amp; AWS.  To do so, you bring up a full cluster
(specifying <code>MULTIZONE=true</code>), and then you add nodes in additional zones
by running <code>kube-up</code> again (specifying <code>KUBE_USE_EXISTING_MASTER=true</code>).</p>

<h3 id="bringing-up-your-cluster">Bringing up your cluster</h3>

<p>Create the cluster as normal, but pass MULTIZONE to tell the cluster to manage multiple zones; creating nodes in us-central1-a.</p>

<p>GCE:</p>
<div class="highlight"><pre style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">curl -sS https://get.k8s.io | <span style="color:#b8860b">MULTIZONE</span><span style="color:#666">=</span><span style="color:#a2f">true</span> <span style="color:#b8860b">KUBERNETES_PROVIDER</span><span style="color:#666">=</span>gce <span style="color:#b8860b">KUBE_GCE_ZONE</span><span style="color:#666">=</span>us-central1-a <span style="color:#b8860b">NUM_NODES</span><span style="color:#666">=</span><span style="color:#666">3</span> bash</code></pre></div>
<p>AWS:</p>
<div class="highlight"><pre style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">curl -sS https://get.k8s.io | <span style="color:#b8860b">MULTIZONE</span><span style="color:#666">=</span><span style="color:#a2f">true</span> <span style="color:#b8860b">KUBERNETES_PROVIDER</span><span style="color:#666">=</span>aws <span style="color:#b8860b">KUBE_AWS_ZONE</span><span style="color:#666">=</span>us-west-2a <span style="color:#b8860b">NUM_NODES</span><span style="color:#666">=</span><span style="color:#666">3</span> bash</code></pre></div>
<p>This step brings up a cluster as normal, still running in a single zone
(but <code>MULTIZONE=true</code> has enabled multi-zone capabilities).</p>

<h3 id="nodes-are-labeled">Nodes are labeled</h3>

<p>View the nodes; you can see that they are labeled with zone information.
They are all in <code>us-central1-a</code> (GCE) or <code>us-west-2a</code> (AWS) so far.  The
labels are <code>failure-domain.beta.kubernetes.io/region</code> for the region,
and <code>failure-domain.beta.kubernetes.io/zone</code> for the zone:</p>
<div class="highlight"><pre style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">&gt; kubectl get nodes --show-labels


NAME                     STATUS                     AGE   VERSION          LABELS
kubernetes-master        Ready,SchedulingDisabled   6m    v1.6.0+fff5156   beta.kubernetes.io/instance-type<span style="color:#666">=</span>n1-standard-1,failure-domain.beta.kubernetes.io/region<span style="color:#666">=</span>us-central1,failure-domain.beta.kubernetes.io/zone<span style="color:#666">=</span>us-central1-a,kubernetes.io/hostname<span style="color:#666">=</span>kubernetes-master
kubernetes-minion-87j9   Ready                      6m    v1.6.0+fff5156   beta.kubernetes.io/instance-type<span style="color:#666">=</span>n1-standard-2,failure-domain.beta.kubernetes.io/region<span style="color:#666">=</span>us-central1,failure-domain.beta.kubernetes.io/zone<span style="color:#666">=</span>us-central1-a,kubernetes.io/hostname<span style="color:#666">=</span>kubernetes-minion-87j9
kubernetes-minion-9vlv   Ready                      6m    v1.6.0+fff5156   beta.kubernetes.io/instance-type<span style="color:#666">=</span>n1-standard-2,failure-domain.beta.kubernetes.io/region<span style="color:#666">=</span>us-central1,failure-domain.beta.kubernetes.io/zone<span style="color:#666">=</span>us-central1-a,kubernetes.io/hostname<span style="color:#666">=</span>kubernetes-minion-9vlv
kubernetes-minion-a12q   Ready                      6m    v1.6.0+fff5156   beta.kubernetes.io/instance-type<span style="color:#666">=</span>n1-standard-2,failure-domain.beta.kubernetes.io/region<span style="color:#666">=</span>us-central1,failure-domain.beta.kubernetes.io/zone<span style="color:#666">=</span>us-central1-a,kubernetes.io/hostname<span style="color:#666">=</span>kubernetes-minion-a12q</code></pre></div>
<h3 id="add-more-nodes-in-a-second-zone">Add more nodes in a second zone</h3>

<p>Let&rsquo;s add another set of nodes to the existing cluster, reusing the
existing master, running in a different zone (us-central1-b or us-west-2b).
We run kube-up again, but by specifying <code>KUBE_USE_EXISTING_MASTER=true</code>
kube-up will not create a new master, but will reuse one that was previously
created instead.</p>

<p>GCE:</p>
<div class="highlight"><pre style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell"><span style="color:#b8860b">KUBE_USE_EXISTING_MASTER</span><span style="color:#666">=</span><span style="color:#a2f">true</span> <span style="color:#b8860b">MULTIZONE</span><span style="color:#666">=</span><span style="color:#a2f">true</span> <span style="color:#b8860b">KUBERNETES_PROVIDER</span><span style="color:#666">=</span>gce <span style="color:#b8860b">KUBE_GCE_ZONE</span><span style="color:#666">=</span>us-central1-b <span style="color:#b8860b">NUM_NODES</span><span style="color:#666">=</span><span style="color:#666">3</span> kubernetes/cluster/kube-up.sh</code></pre></div>
<p>On AWS we also need to specify the network CIDR for the additional
subnet, along with the master internal IP address:</p>
<div class="highlight"><pre style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell"><span style="color:#b8860b">KUBE_USE_EXISTING_MASTER</span><span style="color:#666">=</span><span style="color:#a2f">true</span> <span style="color:#b8860b">MULTIZONE</span><span style="color:#666">=</span><span style="color:#a2f">true</span> <span style="color:#b8860b">KUBERNETES_PROVIDER</span><span style="color:#666">=</span>aws <span style="color:#b8860b">KUBE_AWS_ZONE</span><span style="color:#666">=</span>us-west-2b <span style="color:#b8860b">NUM_NODES</span><span style="color:#666">=</span><span style="color:#666">3</span> <span style="color:#b8860b">KUBE_SUBNET_CIDR</span><span style="color:#666">=</span><span style="color:#666">172</span>.20.1.0/24 <span style="color:#b8860b">MASTER_INTERNAL_IP</span><span style="color:#666">=</span><span style="color:#666">172</span>.20.0.9 kubernetes/cluster/kube-up.sh</code></pre></div>
<p>View the nodes again; 3 more nodes should have launched and be tagged
in us-central1-b:</p>
<div class="highlight"><pre style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">&gt; kubectl get nodes --show-labels

NAME                     STATUS                     AGE   VERSION           LABELS
kubernetes-master        Ready,SchedulingDisabled   16m   v1.6.0+fff5156    beta.kubernetes.io/instance-type<span style="color:#666">=</span>n1-standard-1,failure-domain.beta.kubernetes.io/region<span style="color:#666">=</span>us-central1,failure-domain.beta.kubernetes.io/zone<span style="color:#666">=</span>us-central1-a,kubernetes.io/hostname<span style="color:#666">=</span>kubernetes-master
kubernetes-minion-281d   Ready                      2m    v1.6.0+fff5156    beta.kubernetes.io/instance-type<span style="color:#666">=</span>n1-standard-2,failure-domain.beta.kubernetes.io/region<span style="color:#666">=</span>us-central1,failure-domain.beta.kubernetes.io/zone<span style="color:#666">=</span>us-central1-b,kubernetes.io/hostname<span style="color:#666">=</span>kubernetes-minion-281d
kubernetes-minion-87j9   Ready                      16m   v1.6.0+fff5156    beta.kubernetes.io/instance-type<span style="color:#666">=</span>n1-standard-2,failure-domain.beta.kubernetes.io/region<span style="color:#666">=</span>us-central1,failure-domain.beta.kubernetes.io/zone<span style="color:#666">=</span>us-central1-a,kubernetes.io/hostname<span style="color:#666">=</span>kubernetes-minion-87j9
kubernetes-minion-9vlv   Ready                      16m   v1.6.0+fff5156    beta.kubernetes.io/instance-type<span style="color:#666">=</span>n1-standard-2,failure-domain.beta.kubernetes.io/region<span style="color:#666">=</span>us-central1,failure-domain.beta.kubernetes.io/zone<span style="color:#666">=</span>us-central1-a,kubernetes.io/hostname<span style="color:#666">=</span>kubernetes-minion-9vlv
kubernetes-minion-a12q   Ready                      17m   v1.6.0+fff5156    beta.kubernetes.io/instance-type<span style="color:#666">=</span>n1-standard-2,failure-domain.beta.kubernetes.io/region<span style="color:#666">=</span>us-central1,failure-domain.beta.kubernetes.io/zone<span style="color:#666">=</span>us-central1-a,kubernetes.io/hostname<span style="color:#666">=</span>kubernetes-minion-a12q
kubernetes-minion-pp2f   Ready                      2m    v1.6.0+fff5156    beta.kubernetes.io/instance-type<span style="color:#666">=</span>n1-standard-2,failure-domain.beta.kubernetes.io/region<span style="color:#666">=</span>us-central1,failure-domain.beta.kubernetes.io/zone<span style="color:#666">=</span>us-central1-b,kubernetes.io/hostname<span style="color:#666">=</span>kubernetes-minion-pp2f
kubernetes-minion-wf8i   Ready                      2m    v1.6.0+fff5156    beta.kubernetes.io/instance-type<span style="color:#666">=</span>n1-standard-2,failure-domain.beta.kubernetes.io/region<span style="color:#666">=</span>us-central1,failure-domain.beta.kubernetes.io/zone<span style="color:#666">=</span>us-central1-b,kubernetes.io/hostname<span style="color:#666">=</span>kubernetes-minion-wf8i</code></pre></div>
<h3 id="volume-affinity">Volume affinity</h3>

<p>Create a volume using the dynamic volume creation (only PersistentVolumes are supported for zone affinity):</p>
<div class="highlight"><pre style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-json" data-lang="json"><span style="">kubectl</span> <span style="">create</span> <span style="">-f</span> <span style="">-</span> <span style="">&lt;&lt;EOF</span>
{
  <span style="color:#008000;font-weight:bold">&#34;kind&#34;</span>: <span style="color:#b44">&#34;PersistentVolumeClaim&#34;</span>,
  <span style="color:#008000;font-weight:bold">&#34;apiVersion&#34;</span>: <span style="color:#b44">&#34;v1&#34;</span>,
  <span style="color:#008000;font-weight:bold">&#34;metadata&#34;</span>: {
    <span style="color:#008000;font-weight:bold">&#34;name&#34;</span>: <span style="color:#b44">&#34;claim1&#34;</span>,
    <span style="color:#008000;font-weight:bold">&#34;annotations&#34;</span>: {
        <span style="color:#008000;font-weight:bold">&#34;volume.alpha.kubernetes.io/storage-class&#34;</span>: <span style="color:#b44">&#34;foo&#34;</span>
    }
  },
  <span style="color:#008000;font-weight:bold">&#34;spec&#34;</span>: {
    <span style="color:#008000;font-weight:bold">&#34;accessModes&#34;</span>: [
      <span style="color:#b44">&#34;ReadWriteOnce&#34;</span>
    ],
    <span style="color:#008000;font-weight:bold">&#34;resources&#34;</span>: {
      <span style="color:#008000;font-weight:bold">&#34;requests&#34;</span>: {
        <span style="color:#008000;font-weight:bold">&#34;storage&#34;</span>: <span style="color:#b44">&#34;5Gi&#34;</span>
      }
    }
  }
}
<span style="">EOF</span></code></pre></div>
<p><strong>NOTE:</strong> For version 1.3+ Kubernetes will distribute dynamic PV claims across
the configured zones. For version 1.2, dynamic persistent volumes were
always created in the zone of the cluster master
(here us-central1-a / us-west-2a); that issue
(<a href="https://github.com/kubernetes/kubernetes/issues/23330" target="_blank">#23330</a>)
was addressed in 1.3+.</p>

<p>Now lets validate that Kubernetes automatically labeled the zone &amp; region the PV was created in.</p>
<div class="highlight"><pre style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">&gt; kubectl get pv --show-labels
NAME           CAPACITY   ACCESSMODES   STATUS    CLAIM            REASON    AGE       LABELS
pv-gce-mj4gm   5Gi        RWO           Bound     default/claim1             46s       failure-domain.beta.kubernetes.io/region<span style="color:#666">=</span>us-central1,failure-domain.beta.kubernetes.io/zone<span style="color:#666">=</span>us-central1-a</code></pre></div>
<p>So now we will create a pod that uses the persistent volume claim.
Because GCE PDs / AWS EBS volumes cannot be attached across zones,
this means that this pod can only be created in the same zone as the volume:</p>
<div class="highlight"><pre style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-yaml" data-lang="yaml">kubectl<span style="color:#bbb"> </span>create<span style="color:#bbb"> </span>-f<span style="color:#bbb"> </span>-<span style="color:#bbb"> </span>&lt;&lt;EOF<span style="color:#bbb">
</span><span style="color:#bbb"></span>kind:<span style="color:#bbb"> </span>Pod<span style="color:#bbb">
</span><span style="color:#bbb"></span>apiVersion:<span style="color:#bbb"> </span>v1<span style="color:#bbb">
</span><span style="color:#bbb"></span>metadata:<span style="color:#bbb">
</span><span style="color:#bbb">  </span>name:<span style="color:#bbb"> </span>mypod<span style="color:#bbb">
</span><span style="color:#bbb"></span>spec:<span style="color:#bbb">
</span><span style="color:#bbb">  </span>containers:<span style="color:#bbb">
</span><span style="color:#bbb">    </span>-<span style="color:#bbb"> </span>name:<span style="color:#bbb"> </span>myfrontend<span style="color:#bbb">
</span><span style="color:#bbb">      </span>image:<span style="color:#bbb"> </span>nginx<span style="color:#bbb">
</span><span style="color:#bbb">      </span>volumeMounts:<span style="color:#bbb">
</span><span style="color:#bbb">      </span>-<span style="color:#bbb"> </span>mountPath:<span style="color:#bbb"> </span><span style="color:#b44">&#34;/var/www/html&#34;</span><span style="color:#bbb">
</span><span style="color:#bbb">        </span>name:<span style="color:#bbb"> </span>mypd<span style="color:#bbb">
</span><span style="color:#bbb">  </span>volumes:<span style="color:#bbb">
</span><span style="color:#bbb">    </span>-<span style="color:#bbb"> </span>name:<span style="color:#bbb"> </span>mypd<span style="color:#bbb">
</span><span style="color:#bbb">      </span>persistentVolumeClaim:<span style="color:#bbb">
</span><span style="color:#bbb">        </span>claimName:<span style="color:#bbb"> </span>claim1<span style="color:#bbb">
</span><span style="color:#bbb"></span>EOF</code></pre></div>
<p>Note that the pod was automatically created in the same zone as the volume, as
cross-zone attachments are not generally permitted by cloud providers:</p>
<div class="highlight"><pre style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">&gt; kubectl describe pod mypod | grep Node
Node:        kubernetes-minion-9vlv/10.240.0.5
&gt; kubectl get node kubernetes-minion-9vlv --show-labels
NAME                     STATUS    AGE    VERSION          LABELS
kubernetes-minion-9vlv   Ready     22m    v1.6.0+fff5156   beta.kubernetes.io/instance-type<span style="color:#666">=</span>n1-standard-2,failure-domain.beta.kubernetes.io/region<span style="color:#666">=</span>us-central1,failure-domain.beta.kubernetes.io/zone<span style="color:#666">=</span>us-central1-a,kubernetes.io/hostname<span style="color:#666">=</span>kubernetes-minion-9vlv</code></pre></div>
<h3 id="pods-are-spread-across-zones">Pods are spread across zones</h3>

<p>Pods in a replication controller or service are automatically spread
across zones.  First, let&rsquo;s launch more nodes in a third zone:</p>

<p>GCE:</p>
<div class="highlight"><pre style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell"><span style="color:#b8860b">KUBE_USE_EXISTING_MASTER</span><span style="color:#666">=</span><span style="color:#a2f">true</span> <span style="color:#b8860b">MULTIZONE</span><span style="color:#666">=</span><span style="color:#a2f">true</span> <span style="color:#b8860b">KUBERNETES_PROVIDER</span><span style="color:#666">=</span>gce <span style="color:#b8860b">KUBE_GCE_ZONE</span><span style="color:#666">=</span>us-central1-f <span style="color:#b8860b">NUM_NODES</span><span style="color:#666">=</span><span style="color:#666">3</span> kubernetes/cluster/kube-up.sh</code></pre></div>
<p>AWS:</p>
<div class="highlight"><pre style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell"><span style="color:#b8860b">KUBE_USE_EXISTING_MASTER</span><span style="color:#666">=</span><span style="color:#a2f">true</span> <span style="color:#b8860b">MULTIZONE</span><span style="color:#666">=</span><span style="color:#a2f">true</span> <span style="color:#b8860b">KUBERNETES_PROVIDER</span><span style="color:#666">=</span>aws <span style="color:#b8860b">KUBE_AWS_ZONE</span><span style="color:#666">=</span>us-west-2c <span style="color:#b8860b">NUM_NODES</span><span style="color:#666">=</span><span style="color:#666">3</span> <span style="color:#b8860b">KUBE_SUBNET_CIDR</span><span style="color:#666">=</span><span style="color:#666">172</span>.20.2.0/24 <span style="color:#b8860b">MASTER_INTERNAL_IP</span><span style="color:#666">=</span><span style="color:#666">172</span>.20.0.9 kubernetes/cluster/kube-up.sh</code></pre></div>
<p>Verify that you now have nodes in 3 zones:</p>
<div class="highlight"><pre style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">kubectl get nodes --show-labels</code></pre></div>
<p>Create the guestbook-go example, which includes an RC of size 3, running a simple web app:</p>
<div class="highlight"><pre style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">find kubernetes/examples/guestbook-go/ -name <span style="color:#b44">&#39;*.json&#39;</span> | xargs -I <span style="color:#666">{}</span> kubectl create -f <span style="color:#666">{}</span></code></pre></div>
<p>The pods should be spread across all 3 zones:</p>
<div class="highlight"><pre style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">&gt;  kubectl describe pod -l <span style="color:#b8860b">app</span><span style="color:#666">=</span>guestbook | grep Node
Node:        kubernetes-minion-9vlv/10.240.0.5
Node:        kubernetes-minion-281d/10.240.0.8
Node:        kubernetes-minion-olsh/10.240.0.11

 &gt; kubectl get node kubernetes-minion-9vlv kubernetes-minion-281d kubernetes-minion-olsh --show-labels
NAME                     STATUS    AGE    VERSION          LABELS
kubernetes-minion-9vlv   Ready     34m    v1.6.0+fff5156   beta.kubernetes.io/instance-type<span style="color:#666">=</span>n1-standard-2,failure-domain.beta.kubernetes.io/region<span style="color:#666">=</span>us-central1,failure-domain.beta.kubernetes.io/zone<span style="color:#666">=</span>us-central1-a,kubernetes.io/hostname<span style="color:#666">=</span>kubernetes-minion-9vlv
kubernetes-minion-281d   Ready     20m    v1.6.0+fff5156   beta.kubernetes.io/instance-type<span style="color:#666">=</span>n1-standard-2,failure-domain.beta.kubernetes.io/region<span style="color:#666">=</span>us-central1,failure-domain.beta.kubernetes.io/zone<span style="color:#666">=</span>us-central1-b,kubernetes.io/hostname<span style="color:#666">=</span>kubernetes-minion-281d
kubernetes-minion-olsh   Ready     3m     v1.6.0+fff5156   beta.kubernetes.io/instance-type<span style="color:#666">=</span>n1-standard-2,failure-domain.beta.kubernetes.io/region<span style="color:#666">=</span>us-central1,failure-domain.beta.kubernetes.io/zone<span style="color:#666">=</span>us-central1-f,kubernetes.io/hostname<span style="color:#666">=</span>kubernetes-minion-olsh</code></pre></div>
<p>Load-balancers span all zones in a cluster; the guestbook-go example
includes an example load-balanced service:</p>
<div class="highlight"><pre style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">&gt; kubectl describe service guestbook | grep LoadBalancer.Ingress
LoadBalancer Ingress:   <span style="color:#666">130</span>.211.126.21

&gt; <span style="color:#b8860b">ip</span><span style="color:#666">=</span><span style="color:#666">130</span>.211.126.21

&gt; curl -s http://<span style="color:#b68;font-weight:bold">${</span><span style="color:#b8860b">ip</span><span style="color:#b68;font-weight:bold">}</span>:3000/env | grep HOSTNAME
  <span style="color:#b44">&#34;HOSTNAME&#34;</span>: <span style="color:#b44">&#34;guestbook-44sep&#34;</span>,

&gt; <span style="color:#666">(</span><span style="color:#a2f;font-weight:bold">for</span> i in <span style="color:#b44">`</span>seq <span style="color:#666">20</span><span style="color:#b44">`</span>; <span style="color:#a2f;font-weight:bold">do</span> curl -s http://<span style="color:#b68;font-weight:bold">${</span><span style="color:#b8860b">ip</span><span style="color:#b68;font-weight:bold">}</span>:3000/env | grep HOSTNAME; <span style="color:#a2f;font-weight:bold">done</span><span style="color:#666">)</span>  | sort | uniq
  <span style="color:#b44">&#34;HOSTNAME&#34;</span>: <span style="color:#b44">&#34;guestbook-44sep&#34;</span>,
  <span style="color:#b44">&#34;HOSTNAME&#34;</span>: <span style="color:#b44">&#34;guestbook-hum5n&#34;</span>,
  <span style="color:#b44">&#34;HOSTNAME&#34;</span>: <span style="color:#b44">&#34;guestbook-ppm40&#34;</span>,</code></pre></div>
<p>The load balancer correctly targets all the pods, even though they are in multiple zones.</p>

<h3 id="shutting-down-the-cluster">Shutting down the cluster</h3>

<p>When you&rsquo;re done, clean up:</p>

<p>GCE:</p>
<div class="highlight"><pre style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell"><span style="color:#b8860b">KUBERNETES_PROVIDER</span><span style="color:#666">=</span>gce <span style="color:#b8860b">KUBE_USE_EXISTING_MASTER</span><span style="color:#666">=</span><span style="color:#a2f">true</span> <span style="color:#b8860b">KUBE_GCE_ZONE</span><span style="color:#666">=</span>us-central1-f kubernetes/cluster/kube-down.sh
<span style="color:#b8860b">KUBERNETES_PROVIDER</span><span style="color:#666">=</span>gce <span style="color:#b8860b">KUBE_USE_EXISTING_MASTER</span><span style="color:#666">=</span><span style="color:#a2f">true</span> <span style="color:#b8860b">KUBE_GCE_ZONE</span><span style="color:#666">=</span>us-central1-b kubernetes/cluster/kube-down.sh
<span style="color:#b8860b">KUBERNETES_PROVIDER</span><span style="color:#666">=</span>gce <span style="color:#b8860b">KUBE_GCE_ZONE</span><span style="color:#666">=</span>us-central1-a kubernetes/cluster/kube-down.sh</code></pre></div>
<p>AWS:</p>
<div class="highlight"><pre style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell"><span style="color:#b8860b">KUBERNETES_PROVIDER</span><span style="color:#666">=</span>aws <span style="color:#b8860b">KUBE_USE_EXISTING_MASTER</span><span style="color:#666">=</span><span style="color:#a2f">true</span> <span style="color:#b8860b">KUBE_AWS_ZONE</span><span style="color:#666">=</span>us-west-2c kubernetes/cluster/kube-down.sh
<span style="color:#b8860b">KUBERNETES_PROVIDER</span><span style="color:#666">=</span>aws <span style="color:#b8860b">KUBE_USE_EXISTING_MASTER</span><span style="color:#666">=</span><span style="color:#a2f">true</span> <span style="color:#b8860b">KUBE_AWS_ZONE</span><span style="color:#666">=</span>us-west-2b kubernetes/cluster/kube-down.sh
<span style="color:#b8860b">KUBERNETES_PROVIDER</span><span style="color:#666">=</span>aws <span style="color:#b8860b">KUBE_AWS_ZONE</span><span style="color:#666">=</span>us-west-2a kubernetes/cluster/kube-down.sh</code></pre></div>


				<div class="issue-button-container">
					<p><a href="../multiple-zones.1"><img src="https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/setup/multiple-zones.md?pixel" alt="Analytics" /></a></p>
					
					
					<script type="text/javascript">
					PDRTJS_settings_8345992 = {
					"id" : "8345992",
					"unique_id" : "\/docs\/setup\/multiple-zones\/",
					"title" : "Running in Multiple Zones",
					"permalink" : "https:\/\/kubernetes.io\/docs\/setup\/multiple-zones\/"
					};
					(function(d,c,j){if(!document.getElementById(j)){var pd=d.createElement(c),s;pd.id=j;pd.src=('https:'==document.location.protocol)?'https://polldaddy.com/js/rating/rating.js':'http://i0.poll.fm/js/rating/rating.js';s=document.getElementsByTagName(c)[0];s.parentNode.insertBefore(pd,s);}}(document,'script','pd-rating-js'));
					</script>
					<a href="../multiple-zones.1" onclick="window.open('https://github.com/kubernetes/website/issues/new?title=Issue%20with%20' +
					'k8s.io'+window.location.pathname)" class="button issue">Create an Issue</a>
					
					
					
					<a href="../../editdocs#docs/setup/multiple-zones.md" class="button issue">Edit this Page</a>
					
				</div>
			</div>
		</section>
		<footer>
    <main class="light-text">
        <nav>
            
            
            
            <a href="../../home.1">Documentation</a>
            
            <a href="../../../blog/index.html">Blog</a>
            
            <a href="../../../partners/index.html">Partners</a>
            
            <a href="../../../community/index.html">Community</a>
            
            <a href="../../../case-studies/index.html">Case Studies</a>
            
        </nav>
        <div class="social">
            <div>
                <a href="https://twitter.com/kubernetesio" class="twitter"><span>twitter</span></a>
                <a href="https://github.com/kubernetes/kubernetes" class="github"><span>Github</span></a>
                <a href="http://slack.k8s.io/" class="slack"><span>Slack</span></a>
            </div>
            <div>
                <a href="http://stackoverflow.com/questions/tagged/kubernetes" class="stack-overflow"><span>Stack Overflow</span></a>
                <a href="https://discuss.kubernetes.io" class="mailing-list"><span>Forum</span></a>
                <a href="https://calendar.google.com/calendar/embed?src=nt2tcnbtbied3l6gi2h29slvc0%40group.calendar.google.com" class="calendar"><span>Events Calendar</span></a>
            </div>
            <div>
                <a href="../../getting-started-guides/index.html" class="button">Get Kubernetes</a>
                <a href="https://git.k8s.io/community/contributors/guide" class="button">Contribute</a>
            </div>
        </div>
        <div id="miceType" class="center">
            &copy; 2018 The Kubernetes Authors | Documentation Distributed under <a href="https://git.k8s.io/website/LICENSE" class="light-text">CC BY 4.0</a>
        </div>
        <div id="miceType" class="center">
            Copyright &copy; 2018 The Linux Foundation&reg;. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our <a href="https://www.linuxfoundation.org/trademark-usage" class="light-text">Trademark Usage page</a>
        </div>
    </main>
</footer>

		<button class="flyout-button" onclick="kub.toggleToc()"></button>

<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
    (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-36037335-10', 'auto');
ga('send', 'pageview');


(function () {
    window.addEventListener('DOMContentLoaded', init)

        
        function init() {
            window.removeEventListener('DOMContentLoaded', init)
                hideNav()
        }

    function hideNav(toc){
        if (!toc) toc = document.querySelector('#docsToc')
        if (!toc) return
            var container = toc.querySelector('.container')

                
                if (container) {
                    if (container.childElementCount === 0 || toc.querySelectorAll('a.item').length === 1) {
                        toc.style.display = 'none'
                            document.getElementById('docsContent').style.width = '100%'
                    }
                } else {
                    requestAnimationFrame(function () {
                        hideNav(toc)
                    })
                }
    }
})();
</script>



	</body>
</html>